There are a bunch of options here, and usually in the past I've done something like generating a UDP send/receive sort of setup. However, a few people have pointed out Redis as an interesting option, and I'm kind of liking it.
From the website http://redis.io:
Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.
So that's fairly overkill for what we need, but it turns out that if we turn off persistence, it's a pretty clean solution to what we want. The idea is that you run a server locally, and all your processes just talk with it. It's got good bindings for Python and is nice and fast (it's meant for lots and lots of small messages)
On Ubuntu, you can just do sudo apt-get install redis-server. This will install it such that the server will automatically start running in the background. You can configure the server by editing /etc/redis/redis.conf and then doing sudo service redis-server restart.
Alternatively, you can build it from source by following these instructions http://redis.io/download#installation:
wget http://download.redis.io/releases/redis-stable.tar.gztar xzf redis-stable.tar.gzcd redis-stablemakeAnd now run it with
src/redis-server(I prefer the second approach, just because I like only having the server running when I want it and I forget to manually shut down the server with redis-cli shutdown).
For windows users, grab the latest installer from https://github.com/MSOpenTech/redis/releases
Finally, no matter what you do, you'll probably want the Python bindings for redis, which you can get with
pip install redis
We don't quite want the default settings for redis, so we need to make some small changes. We do this through a redis.conf file. This will either already exist (if you installed it with apt-get), in which case append these commands to the end of the file, or just make a new text file with just these commands in it.
The main thing to do is turn off persistence. By default, redis dumps all of its information to a file every now and then so that it's possible to do backup recovery. We don't want this for our case, so we add this line to the end of the redis.conf file:
save ""
That's all we definitely have to do, but we may also want to allow remote connections. If you're in a situation where multiple computers are accessing the data (i.e. it's not all just local processes on your computer), then add this line:
bind 0.0.0.0
If you're making your own config file, you can start the Redis server by running
redis-server redis.conf
In [2]:
import redis
r = redis.StrictRedis(host='localhost')
r.set('key', 'value')
print r.get('key')
The simplest thing we can do with the Redis server is to set and get key values. The keys are strings and the values are strings, so you can do whatever you want here. Here's a quick example of using it to have two different Nengo models talk to each other. If you start both of these models running, the value in the first one is sent to the second one.
In [3]:
import nengo
import numpy as np
r = redis.StrictRedis(host='localhost')
model1 = nengo.Network()
with model1:
stim = nengo.Node(np.sin)
a = nengo.Ensemble(100, 1)
output = nengo.Node(lambda t, x: r.set('decoded_value', x[0]), size_in=1)
nengo.Connection(stim, a)
nengo.Connection(a, output)
import nengo_gui.ipython
nengo_gui.ipython.IPythonViz(model1, 'model1.cfg')
In [4]:
import nengo
import numpy as np
r2 = redis.StrictRedis(host='localhost')
model2 = nengo.Network()
with model2:
reader = nengo.Node(lambda t: float(r2.get('decoded_value')))
a = nengo.Ensemble(100, 1)
nengo.Connection(reader, a)
import nengo_gui.ipython
nengo_gui.ipython.IPythonViz(model2, 'model2.cfg')
If we start using this a lot, we might start running into problems with key collisions if two different models are running at the same time, both using the same key value. So it's a good idea to not use a generic key like decoded_value and instead introduce some sort of namespace. For example, you might call your keys mymodel.arm_angle and mymodel.velocity, where the first part is some identifier for your model.
Redis has a range of other data types for various use cases. There's a good overview at http://redis.io/topics/data-types-intro.
- Binary-safe strings.
- Lists: collections of string elements sorted according to the order of insertion. They are basically linked lists.
- Sets: collections of unique, unsorted string elements.
- Sorted sets, similar to Sets but where every string element is associated to a floating number value, called score. The elements are always taken sorted by their score, so unlike Sets it is possible to retrieve a range of elements (for example you may ask: give me the top 10, or the bottom 10).
- Hashes, which are maps composed of fields associated with values. Both the field and the value are strings. This is very similar to Ruby or Python hashes.
- Bit arrays (or simply bitmaps): it is possible, using special commands, to handle String values like an array of bits: you can set and clear individual bits, count all the bits set to 1, find the first set or unset bit, and so forth.
- HyperLogLogs: this is a probabilistic data structure which is used in order to estimate the cardinality of a set.
Mostly, I think the string is the main one we'd use, although I could also see the bitarray being handy. Notice that there is no array type, unfortunately.
Redis also supports allowing systems to subscribe to a list of keys and get those changes automatically. I haven't played with that at all, though, but it seems like it'd be handy. Also, there's nice ways to do produce/consume for syncronization, if we want that.
In [ ]: